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Abstract 

We describe techniques for synthesis and verification of re- 
cursive functional programs over unbounded domains. Our 
techniques build on top of an algorithm for satisfiability 
modulo recursive functions, a framework for deductive syn- 
thesis, and complete synthesis procedures for algebraic data 
types. We present new counterexample-guided algorithms 
for constructing verified programs. We have implemented 
these algorithms in an integrated environment for interac- 
tive verification and synthesis from relational specifications. 
Our system was able to synthesize a number of useful recur- 
sive functions that manipulate unbounded numbers and data 
structures. 

1. Introduction 

Software construction is a difficult problem-solving activity. 
It remains a largely manual effort today, despite significant 
progress in software development environments and tools. 
The development becomes even more difficult when the goal 
is to deliver verified software, which must satisfy specifica- 
tions such as assertions, pre-conditions, and post-conditions. 
We believe that quick feedback and error reports are 
essential for practical verification. Verifying programs af- 
ter t hey have been developed is extremely time-consuming 
and it is difficult to argue its cost-effectiveness. 
Our research therefore explores approaches that support in- 
tegrated software construction and verification. An impor- 
tant aspect of such approaches are modular verification tech- 
niques which can check that a function conforms to its local 
specification. In such approach, the verification of an indi- 
vidual function against its specification can start before the 
entire software system is completed, so tools can provide 
rapid feedback that allows specifications and implementa- 
tions to be developed simultaneously. Quoting yD, who re- 
port on the experience with Spec#, 

"If verification ever makes it into the daily rhythm of 
mainstream programming, it will be through a design- 
time interface providing online verification." 

We choose a functional language as the core language for 
the development of verified software. Functional languages 
strike an appealing balance between executability and ver- 
ifiability, predicted already in 112 811 . Although the problem 
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of delivering verified software has been explored through 
a number of different approaches, in a number of success- 
ful cases the development relies heavily on a functional lan- 
guage. In some cases 11911 the researchers have even written 
the entire software system once in a functional language for 
verifiability, and once in a lower-level language for execu- 
tion efficiency. 

Based on the ideas of suitability of a functional paradigm 
and the importance of rapid feedback, we have developed 
a verifier that quickly detects errors in functional programs 
and reports concrete counterexamples, but can also prove the 
correctness of programs ll42l - l44ll . Furthermore, we have in- 
tegrated such countexample-generating verifier into a web- 
browser-based IDE, resulting in a tool for convenient devel- 
opment of verified functional programs. This verifier is the 
starting point of the tool we present in this paper 

Moving beyond verification, we believe that a produc- 
tive development of verified software requires techniques 
for synthesis from specifications. Specifications in terms of 
properties generalize existing declarative programming lan- 
guage paradigms by allowing the statement of constraints 
between inputs and outputs as opposed to always specifying 
outputs as functions of inputs 1 1 3i 120(1 . UnUke deterministic 
specifications, constraints can be composed using conjunc- 
tions, which enables description of the problem as a combi- 
nation of orthogonal requirements. 

This paper introduces synthesis algorithms, techniques 
and tools that integrate synthesis into the development pro- 
cess for functional program. We present a synthesizer that 
can construct the bodies of functions starting solely from 
their postconditions. The programs that our synthesizer pro- 
duces typically manipulate unbounded data types, such as 
algebraic data types and unbounded integers. Thanks to the 
use of deductive synthesis and the availability of a verifier, 
when the synthesizer succeeds, the generated code is guar- 
anteed to be correct for all possible input values. 

Our synthesizer uses specifications as the description of 
the synthesis problems. While it could additionally accept 
input/output examples to illustrate the desired functionality, 
we view such illustration as a special form of input/output 
relation: whereas input/output examples correspond to tests 
and provide a description of a finite portion of the desired 
functionality, we primarily focus on symbolic descriptions. 
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which ensure the desired behavior over an arbitrarily large 
or even infinite domain. From such descriptions, our syn- 
thesizer can automatically generate input/output examples 
when needed, but can also use them and transform them di- 
rectly into executable code. 

A notable degree of automation in our synthesizer comes 
from synthesis procedures 11 12[|2l[ 12211 . which compile spec- 
ification fragments expressed in decidable logics. Our work 
is the first implementation of the synthesis procedure for al- 
gebraic data types ll42ll . 

Note however, that, to capture a variety of scenarios in 
software development, we also support the general problem 
of synthesis of Turing-complete programs. The result is a 
framework for cost-guided application of deductive synthe- 
sis rules, which decompose the problems into subproblems. 

Our synthesizer tightly cooperates with the underlying 
verifier, which allows it to achieve orders of magnitude bet- 
ter performance than using simpler generate-and-test ap- 
proaches. Techniques we use include symbolic transforma- 
tion based on synthesis procedures, as well as synthesis of 
recursive functions using counterexample-guided strategies. 
We have evaluated a number of system architectures and 
trade-offs between symboUc and concrete reasoning in our 
implementation and arrived at an implementation that is suc- 
cessful despite the large space of possible programs. 

We believe that we have achieved a new level of automa- 
tion for a broad domain of recursive functional programs. 
We consider a particular strength of our system that it can 
synthesize code that satisfies a given relational specification 
for all values of inputs, and not only given input/output pairs. 

Despite aiming at a high automation level, we are aware 
that any general-purpose automated synthesis procedure will 
ultimately face limitations of scalability and the ability to 
control the development process. We deployed the synthesis 
algorithm as an interactive assistance that allows the devel- 
oper to interleave manual and automated development steps. 
In our system, the developer can decompose a function and 
leave the subcomponents to the synthesizer, or, conversely, 
the synthesizer can decompose the problem, solve some of 
the subproblems, and leave the remaining open cases for the 
developer To facilitate such synergy, we deploy an anytime 
synthesis procedure, which maintains a ranked list of current 
problem decompositions. The user can interrupt the synthe- 
sizer at any time to display the current solution and continue 
manual development. This is possible thanks to the fact that 
synthesis problems and specification problems are both ex- 
pressed in a unified language based on the construct resem- 
bUng non-deterministic choice. 

1.1 Contributions 

The overall contribution of this paper is an integrated syn- 
thesis and development system for automated and interac- 
tive development of verified programs. A number of tech- 
niques from deductive and inductive reasoning need to come 
together to make such system usable. 



Verifier. Our automated verification environment is the en 



abler of synthesis. We use SMT solvers, specifically Z3 12911 . 
and a fair function unfolding strategy that is effective for suf- 
ficiently surjective abstractions ||43[ 14411 . We have achieved 
substantial speed-ups of this technique for satisfiable con- 
straints through the use of code generation and fair enu- 
meration of structured values. The improvements in verifi- 
cation and falsification have transferred to the improvements 
in synthesis times. 



Implemented synthesis framework. We developed a de- 
ductive synthesis framework that can accept a given set of 
synthesis rules and applies them according to a cost function. 
The framework accepts 1) a path conditions that encode pro- 
gram context, and 2) a relational specifications. It returns the 
function from inputs to outputs as a solution, as well as any 
necessary strengthening of the precondition needed for the 
function to satisfy the specification. We have deployed the 
framework in a web-browser-based environment with con- 
tinuous compilation and the ability to interrupt the synthesis 
to obtain a partial solution in the form of a new program with 
a possibly simpler synthesis problem. 

Data type synthesis. Within the above framework we have 
implemented rules for synthesis of algebraic data type equa- 
tions and disequations ll42ll . as well as a number of general 
rules for decomposing specifications based on their logical 
structure or case splits on commonly useful conditions. We 
have developed program simplification techniques that post- 
process the generated code and make it more readable. 



Support for recursion schemas and symbolic term genera- 
tors. One of the main strengths in our framework is a new 
form of counterexample-guided synthesis that arises from a 
combination of several rules. 

• A set of built-in recursion schemas can solve a problem 
by generating a fresh recursive function. To ensure weU- 
foundedness we have extended our verifier with termi- 
nation checking, and therefore generate only terminating 
function calls in this rule. 

• To generate bodies of functions, we have symbolic term 
generators that systematically generate well-typed pro- 
grams built from selected set of operators (such as alge- 
braic data type constructors and selectors). To test candi- 
date terms against specifications we use the Leon's veri- 
fier To speed up this search, the rule accumulates previ- 
ously found counterexamples. Moreover, to quickly boot- 
strap the set of examples it uses systematic generators 
that can enumerate in a fair way any finite prefix of a 
countable set of structured values. The falsification of 
generated bodies is done by direct execution of code. For 
this purpose, we have developed a lightweight compiler 
for our subset of Scala into bytecodes, replacing many 
constraint reasoning steps by code execution. 
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Function generation by condition abduction. We also 
present and evaluate an alternative counterexample-guided 
rule tailored towards synthesis of recursive conditional func- 
tions, with the following characteristics. 

• Instead of specialized term evaluators, the rule uses a 
general expression enumerator based on generating all 
expressions of a given type jlOll . This results in a broad 
coverage of expressions that the rule can synthesize. It 
uses a new lazy enumeration algorithm for such expres- 
sions with polynomial-time access to the next term to 
enumerate 12311 . Similarly to the previous rule, it filters 
well-typed expressions using counterexamples generated 
from specifications and previous function candidates, as 
well as based on structured value genererators. 

• The most distinctive aspect of this rule is the handling of 
conditional expressions. The expressions are synthesized 
by collecting relevant terms that satisfy a notable number 
of derived test inputs, and then trying to synthesize pred- 
icates that imply the correctness of candidate terms. This 
is an alternative to relying on existing rules to perform 
splitting on simple conditions. Effectively, the additional 
rule performs abduction of conditions until it covers the 
entire input space with a partition of conditions, where 
each partition is associated with a term. 

Evaluation. We evaluate the current reach of our synthe- 
sizer in fully automated mode by synthesizing functions 
such as those that merge, partition, and sort lists of objects, 
where lists are defined using a general mechanism for alge- 
braic data types. This paper presents a description of all the 
above techniques and a snapshot of our results. We believe 
that the individual techniques are interesting by themselves, 
but we also believe that having a system that combines them 
is essential to understand the potential of these techniques 
in addressing the difficult problem as synthesis. To gain full 
experience of the feeling of such development process, we 
therefore invite the reader to explore the system themselves. 

2. Interactive Synthesis and Verification in 
the Leon System 

We start by illustrating through a series of examples how 
developers can leverage our system to write programs that 
are correct by construction. 

Unary numerals. As a first example, we will consider 
tasks related to Unary numerals. While these examples are 
simple in nature, they illustrate some very important points. 
In particular, they show how, using a combination of veri- 
fication and synthesis, one can program functions manipu- 
lating data types in one representation while specifying the 
operations using an abstract view. 

Consider a standard definition of unary numerals as a re- 
cursive data type, with a base case "zero" and a "successor" 
constructor 



sealed abstract class Num 

case object Z extends Num 

case class S(pred: Num) extends Num 

Because it is more convenient to think of these numerals 
in term of their integer value, we can define an abstraction 
function that computes it: 

def value(n:Num) : Int = (n match { 

case Z => 

case S(p) => 1 + value(p) 
}) ensuring (_ > 0) 

The ensuring clause is Scala notation for a postcondi- 
tion II32II . These postconditions are defined by an anonymous 
function, whose single argument denotes the result of the 
function. The underscore notation is a shorthand for x=>x>0, 
so this annotation simply specifies that the integer represen- 
tation of a unary numeral is never negative, and Leon in- 
stantly proves this simple verification condition. 

Using our (verified) abstraction function, we can start 
specifying operations on unary numerals. Consider for in- 
stance the addition operation. Its contract in terms of the 
value function is clear, so we can write it as: 

def add(x : Num, y : Num) : Num — choose { (r: Num) => 

value(r) == value(x) + value(y) 
} 

Here, choose is a special function defined by Leon to rep- 
resent a computation that needs to be synthesized. Similarly 
to the postcondition, it is defined by an anonymous func- 
tions whose result represents the desired output. Contrary to 
postconditions, though, the function (or synthesis predicate) 
can admit multiple arguments, in which case the synthesized 
program should return a tuple of values of the appropriate 
types. 

Upon invocation of the Leon synthesis component, the 
following recursive implementation is derived: 

def add(x : Num, y : Num) : Num = (x match { 

case Z ^ y 

case S(p) ^ add(p, S(y)) 
}) ensuring(r => value(r) == value(x) + value(y)) 

We can continue expanding on these results, and define a 
synthesis predicate for multiplication: 

def mult(x : Num, y : Num) : Num = choose { (r: Num) => 

value(r) == value(x) * value(y) 
} 

Leveraging the previous results for add, our synthesis algo- 
rithm derives the following program: 

def mult(x : Num, y : Num) : Num = (x match { 

case Z => Z 

case S(p) => add(y, mult(p, y)) 
}) ensuring(r ^ value(r) == value(x) * value(y)) 

Both functions are generated within three seconds. 



On Integrating Deductive Synthesis and Verification Systems 



2013/4/23 



List manipulation. We believe this rapid feedback is 
mandatory when developing from specifications. One rea- 
son is that, since contracts are typically partial, results ob- 
tained from under-specifications can be remote from the de- 
sired output. Thus, a desirable strategy is to rapidly iterate 
and refine specifications until the output matches the expec- 
tations. 

As an example, we will consider the task of synthesizing 
the split function necessary in merge sort. We start from 
a standard recursive definition of lists, and we assume the 
existence of recursive functions computing their size (as a 
non-negative integer), and their content (as a set of integers). 

sealed abstract class List 

case class Cons(iiead: Int, tail: List) extends List 

case object Nil extends List 

def size(lst : List) : Int = . . . 

def content(lst : List) : Set[lnt] = ... 

As a first attempt to synthesize split, we try the following 
specification: 

def split(lst : List) : (List, List) = choose { (r : (List, List)) => 
content(lst) == content(r._l) ++ content(r._2) 

} 

Leon instantly generates the following function which, while 
it satisfies the contract, is not particularly useful: 

def split(lst : List) : (List, List) = (1st, Nil) 

To avoid getting a single list with an empty one, we can 
refine the specification by enforcing that the sizes of the 
resulting lists should not differ by more than one: 

def split(lst : List) : (List, List) = choose { (r : (List, List)) => 
content(lst) == content(r._l) ++ content(r._2) 
M abs(size(r._l)-size(r._2)) < 1 

} 

Again, Leon instantly generates a correct, useless, program: 

def split(lst : List) : (List, List) = (1st, 1st) 

We can further refine the specification by stating that the sum 
of the sizes of the two lists should match the size of the input 
one: 

def split(lst : List) : (List, List) = choose { (r : (List, List)) =► 
content(lst) == content(r._l) ++ content(r._2) 
&& abs(size(r._l) - size(r._2)) < 1 
&& (size(r._l) + size(r._2)) == size(lst) 

} 

We then finally obtain a useful split function: 

def split(lst: List): (List, List) = 1st match { 
case Nil ^ (Nil, Nil) 
case Cons(h, Nil) ^ (Nil, Cons(h, Nil)) 
case Cons(hl, Cons(h2, t2)) ^ 

val r = split(t2) 

(Cons(hl, r._l), Cons(h2, r._2)) 

} 



We observe that in this programming style, users can write 
(or generate) code by conjoining orthogonal requirements, 
such as constraints on the sizes and contents, which are only 
indirectly related. The rapid feedback make it possible to 
go through multiple candidates rapidly, strengthening the 
specification as required. 

Sorting. A typical example of a task that is easier to spec- 
ify than to implement is sorting. We conclude this overview 
of Leon's synthesis capabilities by showing how to derive 
an insertion sorting algorithm. We start from the straightfor- 
ward definition of isSorted, a function that checks whether a 
list is sorted: 

def isSorted(lst : 1st) : Boolean = 1st match { 

case Nil => true 

case Cons(_, Nil) => true 

case Cons(xl, xs ® Cons(x2, _)) ^ xl < x2 kk isSorted(xs) 
} 

Using this function, the problem of sorting can be stated as 
simply as: 

def sort(lst : List): List = choose { (r: List) ^ 

isSorted(r) kk content(r) == content(lst) 
} 

To achieve this goal, we start by specifying the helper func- 
tion insertSorted: 

def insertSorted(lst : List, v: Int): List = { 

require(isSorted(lst)) 

choose { (r: List) => 

isSorted(r) kk content(r) == content(lst) ++ Set(v) 

} 
} 

From this, Leon generates the following solution: 

def insertSorted(lst: List, v: Int): List = { 
require(isSorted(lst)) 
1st match { 
case Nil ^ Cons(v, Nil) 
case Cons(h, tail) => 
val r = insertSorted(t, v) 
if (v > h) Cons(h, r) 
else if (h == v) r 
else Cons(v, Cons(h, t)) 
} 

With the help of this insertion function, we can proceed to 
synthesizing sort with the simple specification mentioned 
above. Within five seconds, Leon generates the following 
implementation of insertion sort: 

def sort(lst : List): List = 1st match { 

case Nil => Nil 

case Cons(h, t) => insertSorted(sort(t), h) 
} 
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3. The Leon Verifier 

The results presented in this paper focus on the synthesis 
component of Leon. The language of Leon is a subset of 
Scala, as illustrated through the examples of Section |2] Be- 
sides integers and user-defined recursive data types, Leon 
supports booleans, sets and maps. 

Solver algorithm. At the core of Leon is an algorithm to 
reason about formulas that include user-defined recursive 
functions, such as size, content, and isSorted in Section |2] 
The algorithm proceeds by iteratively examining longer and 
longer execution traces through the recursive functions. It al- 
ternates between an over-approximation of the executions, 
where only unsatisfiability results can be trusted, and an 
under-approximation, where only satisfiability results can be 
concluded. The status of each approximation is checked us- 
ing the state-of-the-art SMT solver Z3 from Microsoft Re- 
search II29II . The algorithm is a semi-decision procedure, 
meaning that it is theoretically complete for counterexam- 
ples: if a formula is satisfiable, Leon will eventually produce 
a model l44ll . Additionally, the algorithm works as a decision 
procedure for a certain class of formulas (14311 . 

In the past, we have used this core algorithm in the con- 
text of verification ll44ll . but also as part of an experiment in 
providing run-time support for declarative programming us- 
ing constructs similar to choose 12011 . We have in both cases 
found the performance in finding models to be suitable for 
the task at hand. [^ 

Throughout this paper, we will assume the existence of an 
algorithm for deciding formulas containing arbitrary recur- 
sive functions. Whenever completeness is an issue, we will 
mention it and describe the steps to be taken in case of, e.g. 
timeout. 

Compilation-based evaluator. Another component of 
Leon on which we rely in this paper is an interpreter based 
on on-the-fly compilation to the JVM. Function definitions 
are typically compiled once and for all, and can therefore be 
optimized by the JIT compiler This component is used dur- 
ing the search in the core algorithm, to validate models and 
to sometimes optimistically obtain counterexamples. We use 
it to quickly reject candidate programs during synthesis (see 
sections |6]and|7ll. 

Ground term generator. Our system also leverages Leon's 
generator of ground terms and its associated model finder 
Based on a generate-and-test approach, it can generate 
small models for formulas by rapidly and fairly enu- 
merating values of any type. For instance, enumerating 
Lists will produce a stream of values Nil(), Cons(0, Nil()), 
Cons(0, Cons(0, Nil())), Cons(l, Nil()), . . . 



4. Deductive Synthesis Framework 

The approach to synthesis we follow in this paper is to derive 
programs by a succession of independently validated steps. 
In this section, we briefly describe the formal reasoning be- 
hind these constructive steps and provide some illustrative 
examples. A more extended exposition of this formal frame- 
work is available in 1112 1. 



4.1 Synthesis Problems 

A synthesis problem is given by a predicate describing a 
desired relation between a set of input and a set of output 
variables, as well as the context (program point) at which 
the synthesis problem appears. We represent such a problem 
as a quadruple 

{a (n > (/)) x\ 

where: 

• a denotes the set of input variables, 

• X denotes the set of output variables, 

• is the synthesis predicate, and 

• n is the path condition to the synthesis problem. 

The free variables of must be a subset of a U x. The 
path condition denotes a property that holds for input at the 
program point where synthesis is to be performed, and the 
free variables of 11 should therefore be a subset of a. 
As an example, consider the following call to choose: 

def f(a : Int) : Int = { 
if(a > 0) { 

choose((x : Int) ^x>0Ma+x<5) 
} else . . . 

} 

The representation of the corresponding synthesis problem 
is: 

|a (a> Ol>a;> OAa + a:< 5) x] (1) 

4.2 Synthesis Solutions 

We represent a solution to a synthesis problem as a pair 



(PIT) 



where: 



•Pis the precondition, and 

• T is the program term. 

The free variables of both P and T must range over a. The 
intuition is that, whenever the path condition and the precon- 
dition are satisfied, evaluating 0[x 1— > T] should evaluate to 
true, i.e. T are realizers for a solution to 5: in given the 
inputs a. Furthermore, for a solution to be as general as pos- 
sible, the precondition must be as weak as possible. 

Formally, for such a pair to be a solution to a synthesis 
problem, denoted as 



' We should also note that since the publication of 14411 . our engineering 
efforts as well as the progress on Z3 have improved running times by 40%. 



{a (n [> . 



h (P I T) 
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the following two properties must hold: 

• Relation refinement: 

U A P ^ (t)[x ^ T] 

This property states that whenever the path- and precon- 
dition hold, the program T can be used to generate values 
for the output variables x such that the predicate is sat- 
isfied. 

• Domain preservation: 

n A (3S : </)) h -P 

This property states that the precondition P cannot ex- 
clude inputs for which an output would exist such that 
is satisfied. 

As an example, a valid solution to the synthesis problem 
dU is given by: 

(a < 5 I 0) 

The precondition a < 5 characterizes exactly the input 
values for which a solution exists, and for all such values, 
the constant is a valid solution term for x. Note that the 
solution is in general not unique; alternative solutions for 
this particular problem include for instance (a < 5 | 5 — a), 
or (a < 5 I if (a < 5) a + 1 else 0). 

A note on path conditions. Strictly speaking, the inclusion 
of the path condition does not add expressive power to the 
representation of synthesis problems. One can easily verify 
that the space of solution terms for \a (11 \> 0) x\ is isomor- 
phic to the one for [a (true [> 11 A 0) x\.\n the latter case, 
the path condition H, is simply included in the precondition 
of the solution. On the other hand, from the definition it fol- 
lows that if {P I T) is a solution and 11 A P is equivalent 
to n A P' then (P' | T) is also a solution to the synthe- 
sis problem. We can let, P' be, for example, HAP, or, as 
another extreme, 11 ^- P. We have therefore found it con- 
venient in the implementation to explicitly keep track of the 
path conditions and allow freedom in the representation of 
the returned precondition P. 

4.3 Inference Rules for Synthesis 

Building on our correctness criteria for synthesis solutions, 
we now describe inference rules for synthesis. Such rules de- 
scribe relations between synthesis problems, capturing how 
some problems can be solved by reduction to others. We 
have shown in previous work how to design a set of rules to 
ensure completeness of synthesis for a well-specified class 
of formulas, e.g. integer linear arithmetic relations 112 ill or 
simple term algebras lll2ll . In the interest of remaining self- 
contained, we shortly describe some generic rules. We then 
proceed to presenting inference rules which allowed us to 
derive synthesis solutions to problem that go beyond such 
decidable domains. 



The validity of each rule can be established independently 
from its instantiations, or from the context in which they it 
is used. This in turn guarantees that the programs obtained 
by successive applications of validated rules are correct by 
construction. 

Generic reductions. As a first example, consider the rule 
One-point in Figure [T] It intuitively reads as follows; "if 
the predicate of a synthesis problem contains a top-level 
atom of the form xq — t, where xq is an output variable not 
appearing in the term t, then we can solve a simpler problem 
where t is substituted for xq, obtain a solution (P | T) and 
reconstruct a solution for the original one by first computing 
the value for t and then assigning as the result for ccg". 

Another, perhaps simpler, example is given by GROUND 
in Figure [T] This rule simply states that if a synthesis prob- 
lem does not refer to any input variable, then it can be treated 
as a satisfiability problem: any model for the predicate can 
then be used as a ground solution term for x. 

Conditionals. The rules we have seen so far generate 
straight-line, unconditional expressions. In order to synthe- 
size programs that include conditional expressions, we need 
rules such as CASE-SPLIT in Figure [T] The intuition behind 
Case-split is that a disjunction in the synthesis predicate 
can be handled by an if-then-else expression in the synthe- 
sized code, and each subproblem (corresponding to predi- 
cates (pi and (p2 in the rule) can be treated separately. As 
one would expect, the precondition for the final program is 
obtained by taking the disjunction of the preconditions for 
the subproblems. This matches the intuition that the disjunc- 
tive predicate should be realizable if and only if one of its 
disjunct is. Note as well that even though the disjunction is 
symmetrical, in the final program we necessarily privilege 
one branch over the other one. This has the interesting side- 
effect that we can, as shown in the rule, add the negation 
of the precondition Pi to the path condition of the second 
problem. This has the potential of triggering simplifications 
in the solution of (p2- An extreme case being when the first 
precondition is true and the "else" branch becomes unreach- 
able. 

The Case-split rule as we presented it applies to dis- 
junctions in synthesis predicates. We should note that it is 
sometimes desirable to explicitly introduce such disjunc- 
tions. For instance, our system includes rules to introduce 
branching on the equality of two variables, to perform case 
analysis on the types of variables (pattern-matching), etc. 
These rules can be thought of as introducing first a disjunct, 
e.g. a = b\J a^b, then applying CASE-SPLIT. 

Recursion Schemas. We now show an example of an in- 
ference rule that produces a recursive function. A common 
paradigm in functional programming is to perform a compu- 
tation by recursively traversing a structure. The rule LlST- 
REC captures one particular form of such a traversal for the 
List recursive type used in the examples of Section |2] The 
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One-point 



la (n t> 0[a;o ^ t]) x] h (P | t) xq 



s(t) 



Ground 



Al 1= vars((/)) n a = 



LiST-REC 



{a (n [> xo = t A (/)) xo , xl h (P I val S := T; {t , x)) ^""""" |a (n xj)) x\^ (true | 7W) 
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Figure 1. Selected synthesis inference rales. 



goal of the rule is to derive a solution consists of a single in- 
vocation to a recursive function rec. The recursive function 
has the following form: 

def rec(ao, a) = { 
require(n2) 
ao match { 
case Nil ^ Ti 
case Cons(/i, t) => 
val f = rec(t, a) 

} 
} ensuring(r => 4>[x h^ r]) 

where ao is of type List. The function iterates over the list ao 
while preserving the rest of the input variables (the environ- 
ment) a. Observe that its postcondition corresponds exactly 
to the synthesis predicate of the original problem. We now 
go over the premises of the rale in detail: 

• The condition (Hi A P) ==> 112 is necessary to ensure 
that the initial call to rec in the final program will satisfy 
its precondition. 

• The condition 112 [ao '-^ Cons{h,t)] => 112 [ao '— >■ t] 
states that the precondition of rec should be inductive, 
i.e. whenever it holds for a list, it should also hold for its 
tail. This is necessary to ensure that the recursive call will 
satisfy the precondition. 

• The subproblem [a (112 O 0[ao '-^ Nil]) xj corresponds 
to the base case (Nil), and thus does not contain the input 
variable ao. 

• The final subproblem is the most interesting, and corre- 
sponds to the case where ao is a Cons, represented by 
the fresh input variables h and t. Because the recursive 
structure is fixed, we can readily represent the result of 
the invocation rec(t,a) by another fresh variable r. We 
can assume that the postcondition of rec holds for that 
particular call, which we represent in the path condition 
as (j)[ao I— >■ i, X I— J- f]. The rest of the problem is obtained 
by substituting ao for Cons(/i,i) in the path condition and 
in the synthesis predicate. 



5. Exploring the Space of Subproblems 

In the previous section, we described a general formal frame- 
work in which we can describe what constitutes a synthe- 
sis problem and a solution. In particular, we have shown 
how synthesis rules decompose synthesis problems into sub- 
problems. In this section, we describe how we automatically 
search across rale instantiations to derive a complete solu- 
tion to a problem. 

Inference rales are non-deterministic by nature. They jus- 
tify the correctness of a solution, but do not by themselves 
describe how one finds that solution. Our search for a solu- 
tion alternates between considering 1) which rales apply to 
given problems, and 2) which subproblems are generated by 
rule instantiations. 

The task of finding rules that apply to a problem intu- 
itively correspond to finding an inference rule whose con- 
clusion matches the structure of a problem. For instance, 
to apply Ground, the problem needs to mention only out- 
put variables. Similarly, to apply LiST-REC to a problem, it 
needs to contain at least one input variable of type List. 

Computing the subproblems resulting from the applica- 
tion of a rule is in general straightforward, as they corre- 
spond to problems appearing in its premise. The GROUND 
rule, for instance, generates no subproblem, while LiST-REC 
generates two. 

and/or search. To solve one problem, it suffices to find 
a complete derivation from one rule application to that prob- 
lem. However, to fully apply a rule, we need to solve all 
generated subproblems. This corresponds to searching for a 
closed branch in an AND/OR tree ll27ll . 

We now describe the expansion of such a tree using an 
example. Consider the problem of removing a given element 
e from a list a. In our logical notation -using a as an abbre- 
viation for content- the problem is: 

|a , e (true [> q(x) = a{a) \ {e}) xj 

We denote this problem by 1 in the tree of Figure |2] While 
we haven't given an exhaustive list of all rales used in our 
system, it is fair to assume that more than one can apply to 
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Figure 2. An and/or search tree used to illustrate our 
search mechanism. Circles are OR nodes and represent prob- 
lems, while boxes are and nodes and represent our rule ap- 
plications. Nodes in grey are closed (solved). 



this problem. For instance, we could case-split on the type of 
a, or apply LiST-REC to a. We represent these two options 
by A and B respectively in the tree. 

Following the option B and applying LiST-REC with the 
path condition 112 = true trivially satisfies the first two 
premises of the rules, and generates two new problems (5 
and 6). Problem 5 is: 

|e (true [> a{x) = a(Nil) \ {e}) xj 

where the predicate simplifies to a{x) = 0. This makes it 
possible to apply the GROUND rule (node G). This generates 
no subproblem, and closes the subbranch with the solution 
solution (true | Nil). Problem 4 has the form: 

Ir ,h,t,e {a{r) = a{t) \ {e}> 
a{x) ~ a(Cons(/i,i)) \ {e}) x] 

Among the many possible rule applications, we can choose 
to case-split on the equality h = e (node F). This generates 
two subproblems. Problem 6 

fr ,h,t,e {a{r) = a{t) \ {e} A e = h\> 
a{x) = Q:(Cons(/i,t)) \ {e}) xj 

and a similar problem 7, where e ^ h appears in the path 
condition instead of e = h. Both subproblems can be solved 
by using a technique we will describe in Section|6]to derive 
a term satisfying the synthesis predicate, effectively closing 
the complete branch from the root. The solutions for prob- 
lem 6 and 7 are (true | r) and (true | Cons(ft.,r)) respec- 
tively. A complete reconstruction of the solution given by 
the branch in gray yields the program: 

def rec(a : List) : List = a match { 
case Nil => Nil 
case Cons(h,t) => 
val r = rec(t) 
if(e == h) r 
else Cons(h,r) 
} 

In the interest of space, we have only described the 
derivations that lead to the search. In practice of course, not 
all correct steps are taken in the right order The interleaving 
of expansions of AND and OR nodes is driven by the esti- 
mated cost of problems and solutions. 

Cost models. In order to drive the search, we assign to 
each problem and to each rule application an estimated cost, 
which is supposed to under-approximate to actual final cost 
of a closed branch. For OR nodes (problems), the cost is 
simply the minimum of all remaining viable children, while 
for AND nodes (rule applications) we take the sum of the cost 
of each children plus a constant. That constant intuitively 
corresponds to the extra complexity inherent to a particular 
rule. 



On Integrating Deductive Synthesis and Verification Systems 



2013/4/23 



A perfect measure for cost would be the running time 
of the corresponding program. However, this is particularly 
hard to estimate, and valid under-approximations would 
most likely be useless. We chose to measure program size 
instead, as we expect it to be an reasonable proxy for com- 
plexity. We measure the size of the program as the number of 
branches, weighted by their proximity to the root. We found 
this to be have a positive influence on the quality of solu- 
tions, as it discourages near-top-level branching. 

Using this metric, the cost inherent to a rule application 
roughly corresponds to the extra branches it introduces in the 
program. We use a standard algorithm for searching for the 
best solution 12711 . and the search thus always focuses on the 
current most promising solution. In our example in Figure 
we could imagine that after the case split at F, the B branch 
temporarily became less attractive. The search then focuses 
for a while on the A branch, until expansion on that side (for 
instance by case-splitting on the type of the Ust) reached a 
point where the minimal possible solution was worse than 
the B branch. We note that the complete search takes about 
two seconds. 

Anytime synthesis. Because we maintain the search tree 
and know the current minimal solution at all times, we can 
stop the synthesis at any time and obtain a partial program 
that is likely to be good. This option is available in our im- 
plementation, both from the console mode and the web in- 
terface. In such cases, Leon will return a program containing 
new invocations of choose corresponding to the open sub- 
problems. 

6. Symbolic Term Exploration 

In previous sections, we have introduced the notion of syn- 
thesis inference rules, and described how to search over rule 
applications that generate subproblems. In this section, we 
describe one of our most important rules, which is responsi- 
ble for closing most of the branches in search trees. We call 
it Symbolic Term Exploration (STE). 

The core idea behind STE is to symbolically represent 
many possible terms (programs), and to iteratively prune 
them out using counterexamples and test case generation 
until either 1) a valid term is proved to solve the synthesis 
problem or 2) all programs in the search spaces have been 
shown to be inadequate. Since we already have rules that 
take care of introducing branching constructs or recursive 
functions, we focus STE on the search for terms consisting 
only of constructors and calls to existing functions. 

Recursive generators. We start from a universal non- 
deterministic program that captures all the (deterministic) 
programs which we wish to consider as potential solutions. 
We then try to resolve the non-deterministic choices in such 
a way that the program realizes the desired property. Resolv- 
ing the choices consists in fixing some values in the pro- 
gram, which we achieve by running a counterexample driven 
search. 



We describe our non-deterministic programs as a set of 
recursive non-deterministic generators. Intuitively, a gener- 
ator for a given type is a program that produces arbitrary 
values of that type. For instance, a generator for positive in- 
tegers could be given by: 

def genlnt() : Int = if(*) else (1 + genlnt()) 

where • represents a non-deterministic boolean value. Sim- 
ilarly a non-deterministic generator for the List type could 
take the form: 

def genList() : List = if(*) Nil else Cons(genlnt(), genList()) 

It is not required that generators can produce every value 
for a given type; we could hypothesize for instance that our 
synthesis solutions will only need some very specific con- 
stants, such as 0, 1 or —1. What is more likely is that our 
synthesis solutions will need to use input variables and ex- 
isting functions. Our generators therefore typically include 
variables of the proper type that are accessible in the synthe- 
sis environment. Taking these remarks into account, if a and 
b are integer variables in the scope, and f is a function from 
Int to Int, a typical generator for integers would be: 

def genlnt() : Int = if(*) else if(*) 1 else if(*) -1 
else if(*) a else if(*) b else f(genlnt()) 

From generators to formulas. Generators can in principle 
be any function with unresolved non-deterministic choices. 
For the sake of the presentation, we assume that they are 
"flat", that is, they consist of a top-level non-deterministic 
choice between n alternatives. (Note that the examples given 
above all have this form.) 

Encoding a generator into an SMT term is straightfor- 
ward: introduce for each invocation of a generator an un- 
interpreted constant c of the proper type, and for each non- 
deterministic choice as many boolean variables b as there are 
alternatives. Encode that exactly one of the b variables must 
be true, and constrain the value of c using the b variables. 

Recursive invocations of generators can be handled simi- 
larly, by inserting another c variable to represent their value 
and constraining it appropriately. Naturally, these recursive 
instantiations must stop at some point: we then speak of 
an instantiation depth. As an example, the encoding of the 
gen List generator above with an instantiation depth of 1 and 
assuming that genint generates or a is: 

(6i V62)A(-6i V-fea) 
A 6i ^ ci = Nil A &2 ^ ci = Cons(c2,C3) 
A (&3 V 64) A (^63 V ^^4) 
A 63 ^ C2 = A 64 ^ C2 = a 

A (6,5 V fee) A (-65 V -fee) 

A ^5 => C3 = Nil A fee => C3 = Cons(c4,C5) 

A -fee 



On Integrating Deductive Synthesis and Verification Systems 



2013/4/23 



The clauses encode the following possible values for ci: Nil, 
Cons(0, Nil) and Cons(a, Nil). Note the constraint ^6g which 
encodes the instantiation depth of 1, by preventing the values 
beyond that depth (namely C4 and C5) to participate in the 
expression. 

For a given instantiation depth, a valuation for the b vari- 
ables encodes a determinization of the generators, and as a 
consequence a program. We solve for such a program by 
running a refinement loop. 

Refinement loop: discovering programs. Consider a syn- 
thesis problem [a (11 [>(()) S], where we speculate that a 
generator for the types of x can produce a program that real- 
izes 0. We start by encoding the non-deterministic execution 
of the generator for a fixed instantiation depth (typically, we 
start with 0). Using this encoding, the problem has the form; 



(j) /\B{a,h,c) A C{c,x) 



(2) 



where is the synthesis problem, B is the set of clauses 
obtained by encoding the execution of the generator and C 
is a set of equalities tying S to a subset of the c variables. 
Note that by construction, the values for c (and therefore for 
x) are uniquely determined when a and h are fixed. 

We start by finding values for a and h such that (|2]i holds. 
If no such values exist, then our generators at the given 
instantiation depth are not expressive enough to encode a 
solution to the problem. Otherwise, we extract for the model 
the values 60 ■ They describe a candidate program, which we 
put to the test. 

Refinement loop: falsifying programs. We search for a 
solution to the problem: 



^(j) A B{a, bo, c) A C{c, x) 



(3) 



Note that bo are constants, and that c and x are therefore 
uniquely determined by a this intuitively comes from the 
fact that bo encodes a deterministic program, that c encodes 
intermediate values in the execution of that program, and 
that X encodes the result. With this in mind, it becomes clear 
that we are really solving for a. 

If no such a exist, then we have found a program that 
realizes 4> and we are done. If on the other hand we can 
find oo, then this constitutes an input that witnesses that our 
program does not meet the specification. In this case, we can 
discard the program by asserting -^ /\b, and going back to 
0. 

Eventually, because the set of possible assignments to 6 is 
finite (for a given instantiation depth) this terminates. If we 
have not found a program, we can increase the instantiation 
depth and try again. When the maximal depth is reached, we 
give up. 

Filtering with concrete execution. While termination is 
in principle guaranteed just by successive elimination of 
programs in the refinement loop, the formula encoding the 



non-deterministic program typically grows exponentially as 
instantiation depth increases. As the number of programs 
grows, the difficulty for the solver to satisfy (|2) or (|3} also 
increases. As an alternative to symbolic elimination, we can 
often use concrete execution on a set of inputs to rule out 
many programs. We rely on Leon's capabiUty for small 
model finding (see Section O to generate inputs that satisfy 
the path condition. We then use on-the-fly code generation 
to compile the symbolic program into a function that takes 
as arguments the input variables as well as a boolean ar- 
ray encoding the non-deterministic choices. This allows us 
to rapidly discard hundreds or even thousand of programs. 
Whenever the change is substantial, we regenerate a new 
formula for (|2} with much fewer boolean variables and con- 
tinue from there. Note that very often, when STE is applied 
to a problem it cannot solve, concrete execution rules out all 
programs in a fraction of a second and symbolic reasoning 
is never applied. 

7. Type-Driven Counterexample-Guided 
Synthesis with Condition Abduction 

Our second larger rule focuses on synthesizing recursive 
functions that satisfy a given specification. We assume that 
we are given a function header and a postcondition, and that 
we aim to synthesize a recursive function body. Note that 
the expression must be 1) a well-typed term with respect 
to the context of the program and 2) valid according to 
the imposed formal specification. Therefore, an approach 
to solve this kind of synthesis problems could be based on 
searching the space of all expressions that can be built from 
all declarations visible at the corresponding place in the 
program, i.e. in the scope of choose, while limiting attention 
to those that type-check, have the desired type, and satisfy 
the given formal specification. 

An obvious drawback of such approach is that, unless the 
process is carefully guided, the search becomes unfeasible 
due to search space explosion. In practice we indeed found 
that trivial generate-and-test strategies scale poorly with the 
number of visible declarations and the search becomes prac- 
tically unfeasible even for small programs. 

7.1 Condition Abduction 

Our idea for guiding the search and incremental construction 
of correct expressions comes from the area of abductive 
reasoning jm llTll . Abductive reasoning, sometimes also 
called "inference to the best explanation", is a method of 
reasoning in which one chooses a hypothesis that would 
explain the observed evidence in the best way. 

The motivation behind the approach to applying abduc- 
tive reasoning to program synthesis comes from examining 
implementations of practical purely functional, recursive al- 
gorithms. The key observation is that recursive functional al- 
gorithms share a similar pattern. They implement behaviour 
through a combination of case analysis with control flow ex- 



On Integrating Deductive Synthesis and Verification Systems 



10 



2013/4/23 



pressions (e.g. if-then-else) and recursive calls. This pattern 
is encoded with a branching control flow expression that par- 
titions the space of input values such that each branch repre- 
sents a correct implementation for a certain partition. Such 
partitions are defined by conditions that guard branches in 
the control flow. 

This allows synthesizing branches separately by search- 
ing for implementations that evaluate correctly only for cer- 
tain inputs while restricting the search space. Rather than 
speculatively applying CASE-SPLIT rule to obtain subprob- 
lems and finding solutions for each branch by case analy- 
sis (as described in Section HJi, this idea applies a similar 
strategy in the reverse order - getting a candidate program 
and searching for a condition that would make it correct. 
Thus, the idea of abductive reasoning can be applied to guess 
the condition that defines a valid partition, i.e. "abduce" the 
explanation for a partial implementation, with respect to a 
given candidate program. The rule progressively appUes this 
technique and enables effective search and construction of a 
control flow expression that represents a correct implemen- 
tation for more and more input cases, eventually construct- 
ing an expression that is a solution to the synthesis problem. 

7.2 The Algorithm Used in the Rule 

Based on these observations, we present our rule that em- 
ploys a new technique for guiding the search with ranking 
and filtering based on counterexamples, as well as construct- 
ing expressions from partially correct implementations. It is 
presented in Algorithm[T] 

Algorithm 1 Synthesis with condition abduction 
Require: path condition 11, predicate (j>, a collection of 
expressions s t> synthesis problem |a (11 > cf)) S| 

1: p' = true o maintain the current partition 

2: sol = (Xx.x) t> maintain a partial solution 

3: TM = SAMPLEMODELS(a) o set of example models 
4: repeat 

5: get a set of expressions E from s > candidates 

6: for each e in _E do > count passed examples pe for e 
7: Pe = \{m^ Ai \ e{m) is correct}] > evaluate 

8: f ~ arg max ees Pe » the highest ranked expression 
9: if solution (E A p' | f) is vahd then 
10: return (11 | {sol f)) > a solution is found 

11: else 

12: extract new counterexample model m 
13: A4 ~ AiU m o accumulate examples 

14: c = BRANCHSYN(f,p, g, s) > cafl Algorithm |2] 
15: if c ^ FALSE then > a branch is synthesized 

16: sol ~ [\x. {sol (if c then f else x))) 

17: p' = p' A ^c t> update current partition 

18: until s is not empty 

The algorithm applies the idea of abducing conditions to 
progressively synthesize and verify branches of a correct im- 



plementation for an expanding partition of inputs. The input 
to the algorithm is a path condition 11, a predicate (f) (defined 
by synthesis problem |a (11 O 4>) a;|), and a collection of 
expressions s. 

Condition p' defines which inputs are left to consider at 
any given point in the algorithm; these are the inputs that 
belong to the current partition. The initial value of p' is 
true, so the algorithm starts with a partition that covers the 
whole initial input space constrained only by the path con- 
dition n. Let pi, . . . ,pk, where A: > 0, be conditions ab- 
duced up to a certain point in the algorithm. Then p' repre- 
sents the conjunction of negations of abduced conditions, i.e. 
p' — ^pi A ... A -^Pk- Together with the path condition, it 
defines the current partition which includes all input values 
for which there is no condition abduced (nor correct imple- 
mentation found). Thus, the guard condition for the current 
partition is defined by 11 A p'. The algorithm maintains the 
partial solution sol, encoded as a function, sol encodes an 
expression which is correct for all input values that satisfy 
any of the abduced conditions and this expression can be re- 
turned as a partial solution at any point. Additionally, the al- 
gorithm accumulates example models in the set Ai. Ground 
term generator, described in Section |3] is used to construct 
the initial set of models in Al . To construct a model, for each 
variable in a, the algorithm assigns a value sampled from the 
ground term generator Note that more detailed discussion 
on how examples are used to guide the search is deferred to 
SectionO 

The algorithm repeats enumerating all possible expres- 
sions from the given collection until it finds a solution. In 
each iteration, a batch of expressions E is enumerated and 
evaluated on all models from A4. The results of such eval- 
uation are used to rank expressions from E. The algorithm 
considers the expression of the highest rank f as a candidate 
solution and checks it for validity. If f represents a correct 
implementation for the current partition, i.e. if (11 A p' | f) 
is a valid solution, then the expression needed to complete 
a valid control flow expression is found. The algorithm re- 
turns it as solution for which [a (11 > cj)) 5:| h (11 | {sol f)) 
holds. Otherwise, the algorithm extracts the counterexample 
model m, adds it to the set A4, and continues by trying to 
synthesize a branch with expression f (it does so by call- 
ing Algorithm|2]which will be explained later). If Branch- 
Syn returns a valid branch condition, the algorithm updates 
the partial solution to include the additional branch (thus ex- 
tending extending the space of inputs covered by the partial 
solution), and refines the current partition condition. New 
partition condition reduces the synthesis to a subproblem, 
ensuring that the solution in the next iteration covers cases 
where c does not hold. The algorithm eventually, given the 
appropriate terms from s, finds an expression that forms a 
complete correct implementation for the synthesis problem. 

Algorithm|2]tries to synthesize a new branch by abducing 
a valid branch condition c. It does this by enumerating a set 
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Algorithm 2 Synthesize a branch 



Require: expression ?", condition p', predicate q, and a 
collection of expressions s > passed from Algorithm[T] 

1: function BRANCHSYN(f,p',q,s) 

2: Ai' ~ 9 > set of accumulated counterexamples 

3: get a set of expressions £" from s > candidates 

4: for eacli c in E' do 

5: if for eacli model min M\ c{m) ~ false then 

6: if solution (H A c | f) is valid then 

7: return c t> a condition is abduced 

8: else 

9: extract the new counterexample model m 

10: M.' = M.' \Jm t> accumulate counterexamples 

11: return FALSE t> no condition is found 

of expressions E' from s and checking whether it can find a 
valid condition expression, that would guard a partition for 
which the candidate expression f is correct. The algorithm 
accumulates counterexamples models in M' and considers 
a candidate expression c only if it prevents all accumulated 
counterexamples. The algorithm checks this by evaluating c 
on m, i.e. c{m), for each accumulated counterexample m. 
If a candidate expression c is not filtered out, the algorithm 
checks if c represents a valid branch condition, i.e. whether 
(n A c I f) is a valid solution. If yes, the algorithm returns 
c which, together with f, comprises a valid branch in the 
solution to \a (11 \> (jj) x\. Otherwise, it adds a new coun- 
terexample model to M.' and continues with the search. If 
no valid condition is in E' , the algorithm returns FALSE. 

7.3 Organization of the Search 

For getting the collection of expressions s, the rule uses 
term generators that generate all well typed terms accord- 
ing to type constraints derived from the context of a pro- 
gram |110l|23|]. This has the advantage of initial search space 



restriction inherent to the generator that limits enumerated 
expressions only to those that are well typed. The complete- 
ness property of such generators ensures systematic enumer- 
ation of all candidate solutions that are defined by the set 
of given type constraints. For verification, the rule uses the 
Leon verifier, that allows checking validity of expressions 
that are supported by the underlying theories and obtaining 
counterexample models. 

The context of the algorithm as a rule in the Leon synthe- 
sis framework imposes limits on the portion of search space 
explored by each rule instantiation. This allows incremental 
and systematic progress in search space exploration and, due 
to the mixture with other synthesis rules, offers benefits in 
both expressiveness and performance of synthesis. The rule 
offers flexibility in adjusting necessary parameters and thus 
a fine-grain control over the search - for our experiments, the 
size of candidate sets of expressions enumerated in each it- 
eration n is 50 (and is doubled in each iteration) and 20, in 
the case of Algorithm[T|and|2] respectively. 



Using (counter-)examples. A technique that brings signif- 
icant performance improvements when dealing with large 
search spaces is guiding the search and even avoid con- 
sidering candidate expressions according to the information 
from examples generated during synthesis. As described ear- 
lier, after checking an unsatisfiable formula, the rule queries 
Leon for the witness model and accumulates examples that 
are used to narrow down the search space. 

Algorithm 12] uses accumulated counterexamples to filter 
out unnecessary candidate expressions when synthesizing a 
branch. It makes sense to consider a candidate expression 
for a branch condition, c, for a check whether c makes f a 
correct implementation, only if c prevents all accumulated 
counterexamples that akeady witnessed unsatisfiability of 
the correctness formula for f, i.e. if Vtti G Ai'. c -^ ^m. 
Otherwise, if 3m e A^'.^(c —)• -^m), then m is a valid 
counterexample to the verification of (11 A c | f ) . This ef- 
fectively guides the search by the results of previous veri- 
fication failures while filtering out candidates before more 
expensive verification check are made. 

Algorithm [T] uses accumulated models to quickly test 
and rank expressions by evaluating models according to the 
specification. The current set of candidate expressions E is 
evaluated on the set of accumulated examples M and re- 
sults of such evaluation are used to rank the candidates. We 
call an evaluation of a candidate e on a model m correct, if 
m satisfies path condition 11 and the result of the evaluation 
satisfies given predicate q. The algorithm counts the num- 
ber of correct evaluations, ranks the candidates accordingly 
and considers only the candidate of the highest rank. The ra- 
tionale is that the more correct evaluations, the more likely 
the candidate represents a correct implementation for some 
partition of inputs. Note that evaluation results may be used 
only for ranking but not for filtering, because each candidate 
may represent a correct implementation for a certain parti- 
tion of inputs, thus incorrect evaluations are expected even 
for valid candidates. Since the evaluation amounts to execut- 
ing the specification this technique is efficient in guiding the 
search toward correct correct implementations while avoid- 
ing unnecessary verification checks. 

8. Implementation and Results 

We have implemented these techniques in Leon, a system 
for verification and synthesis of functional program, thus 
extending it from the state described in Section |3] Our im- 
plementation and the online interface are available from 



http: //lara. epf 1. ch/leon/ 



The front end to Leon is the standard Scala compiler (for 
Scala 2.9). Scala compiler performs type checking and tasks 
such as the expansion of implicit conversions, from which 
Leon directly benefits. Leon programs also execute as valid 
Scala programs. Leon checks that the syntax trees produced 
conform to the subset that it expects and then performs 
verification and synthesis. 
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Figure 3. We consider a problem as synthesized if the so- 
lution generated is correct after manual inspection. For each 
generated program, we provide the size of its syntax tree and 
the number of function calls it contains. Proved problems are 
those for which the synthesized program can be automati- 
cally proven to match its specification. 

We have developed several interfaces for Leon. Leon 
can be invoked as a batch command-line tool that accepts 
verification and synthesis tasks and outputs the results of the 
requested tasks. If desired, there is also a console mode that 
allows applying synthesis rules in a step-by-step fashion and 
is useful for debugging purposes. 

To facilitate interactive experiments and the use of the 
system in teaching, we have also developed an interface that 
executes in the web browser, using the Play framework of 
Scala as well as JavaScript editors. Our browser-based in- 
terface supports continuous compilation of Scala code, al- 
lows verifying individual functions with a single keystroke 
or click, as well as synthesizing any given choose expres- 
sion. In cases when the synthesis process is interrupted, the 
synthesizer can generate a partial solution that contains a 
program with further occurrences of the choose statement. 

8.1 Results 

In order to evaluate our system, we developed benchmarks 
with reusable abstraction functions. These abstraction func- 
tions allow for a concise specification of each operation 
without requiring any insight on its resulting implementa- 
tion. It is interesting to notice that these functions generally 
abstract any structural invariant inherent to the underlying 
data-structure. For instance, the synthesis of 

def add(a: Num, b: Num) = choose { 

(res: Num) => value(r) == value(a) + value(b) 



} 

would result in vastly different programs depending on the 

implementation of Number. 

Our set of benchmarks displayed in Figure |3] covers the 
synthesis of various operations over custom data-structures 
with invariants, specified through the lens of abstraction 
functions. These benchmarks use specifications with are 
both easy to understand and much shorter than resulting pro- 
grams (except in trivial cases). We believe these are key fac- 
tors in the evaluation of any synthesis procedure. The defini- 
tions and specifications of all the benchmarks can be found 
in appendix. 

Synthesis is performed in order, meaning that an opera- 
tion will be able to reuse all previously synthesized ones, 
thus mimicking the usual development process. 

We can see in Figure |3] the list of programs we success- 
fully synthesized. Each synthesized program has been man- 
ually validated to be a solution that a programmer might ex- 
pect. Our system typically also proves automatically that the 
resulting program matches the specification for all inputs. In 
certain cases, the lack of inductive invariants prevents such 
fully-automated proof, which is a limitation of our verifier. 
Note that we stop verification after a timeout of 3 seconds. 

In almost all cases, the synthesis succeeds sufficiently fast 
for a reasonable interactive experience. 

9. Related Work 



Our approach blends deductive synthesis II25L 126113611 . which 
incorporates transformation of specifications, inductive rea- 
soning, recursion schemes and termination checking, with 
modem SMT techniques and constraint solving for exe- 
cutable constraints. As one of our subroutines we include 
complete functional synthesis for integer linear arithmetic 
1 2M and extend it with complete functional synthesis for al- 
gebraic data types 11121 14211 . This gives us building blocks 
for synthesis of recursion-free code. To synthesize recursive 
code we build on and further advance the counterexample- 
guided approach to synthesis 1371]. 

Deductive synthesis frameworks. Early work on synthesis 
|25L |2q) focused on synthesis using expressive and undecid- 
able logics, such as first-order logic and logic containing the 
induction principle. 

Programming by refinement has been popularized as a 
manual activity 12|, |46[] . Interactive tools have been devel- 
oped to support such techniques in HOL |4J1. A recent exam- 
ple of deductive synthesis and refinement is the Specware 
system from Kesterel 13611 . We were not able to use the sys- 
tem first-hand due to its availability policy, but it appears to 
favor expressive power and control, whereas we favor au- 
tomation. 

A combination of automated and interactive development 
is analogous to the use of automation in interactive theorem 
pro vers, such as Isabelle Bill . However, whereas in verifica- 
tion it is typically the case that the program is available, the 
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emphasis here is on constructing the program itself, starting 
from specifications. 

Work on synthesis from specifications ll40ll resolves some 
of these difficulties by decoupling the problem of inferring 
program control structure and the problem of synthesizing 
the computation along the control edges. The work lever- 
ages verification techniques that use both approximation and 
lattice theoretic search along with decision procedures, but 
appears to require more detailed information about the struc- 
ture of the expected solution than our approach. 

Synthesis with input/output examples. One of the first 
works that addressed synthesis with examples and put induc- 
tive synthesis on a firm theoretical foundation is the one by 
Summers |4l|l. Subsequent work presents extensions of the 
classical approach to induction of functional Lisp-programs 
I 111 llSn . These extensions include synthesizing a set of 



equations (instead of just one), multiple recursive calls and 
systematic introduction of parameters. Our current system 
lifts several restrictions of previous approaches by supprot- 
ing reasoning about arbitrary datatypes, supporting multiple 
parameters in concrete and symbolic I/O examples, and al- 
lowing nested recursive calls and user-defined declarations. 
Inductive (logic) programming that explores automatic 
synthesis of (usually recursive) programs from incomplete 
specifications, most often being input/output examples JTl 
boll , influenced our work. Recent work in the area of pro- 
gramming by demonstration has shown that synthesis from 
examples can be effective in a variety of domains, such as 
spreadsheets 13511 . Advances in the field of SAT and SMT 
solvers inspired counter-example guided iterative synthe- 



sis list |37|], which can derive input and output examples 



from specifications. Our tool uses and advances these tech- 
niques through two new counterexample-guided synthesis 
approaches. 

Synthesis based on finitization techniques. Program 
sketching has demonstrated the practicality of program syn- 
thesis by focusing its use on particular domains 1137143 911 . 
The algorithms employed in sketching are typically focused 
on appropriately guided search over the syntax tree of the 
synthesized program. The tool we presented shows one way 
to move the ideas of sketching towards infinite domains. In 
this generalization we leverage reasoning about equations as 
much as SAT techniques. 

Reactive synthesis. Synthesis of reactive systems gener- 
ates programs that run forever and interact with the envi- 
ronment. However, known complete algorithms for reac- 
tive synthesis work with finite-state systems ll34ll or timed 
systems uJ]. Such techniques have applications to control 
the behavior of hardware and embedded systems or concur- 
rent programs IHSll . These techniques us ually take specifi- 
cations in a fragment of temporal logic 113 3ll and have re- 
sulted in tools that can synthesize useful hardware compo- 
nents lll4lll5ll . Recently such synthesis techniques have been 



extended to repair that preserves good behaviors [|61, which 
is related to our notion of partial programs that have remain- 
ing choose statements. 

10. Conclusions and Analysis 

Software synthesis is a difficult problem but we believe 
it can provide substantial help in software development. 
We have presented a new framework for synthesis that 
combines transformational and counterexample-guided ap- 
proaches. Our implemented system can synthesize and prove 
correct functional programs that manipulate unbounded data 
structures such as algebraic data types. We have used the 
system to synthesized algorithms that manipulate list and 
tree structures. The algorithm can be combined with manual 
transformations or run-time constraint solving to cover the 
cases where static synthesis does not fully solve the prob- 
lem. Our current counterexample-guided synthesis steps are 
domain-agnostic, while somewhat limits their scalability, so 
we expect improved results using domain-specific genera- 
tors, such as the ones used in testing tools UDITA |80 and 
Quickcheck ISfl. Our framework leverages the state of the 
art SMT solving technology and an effective mechanism 
for solving certain classes of recursive functions. Thanks to 
this technology, it was able to synthesize programs over un- 
bounded domains that are guaranteed to be correct for all 
inputs. 
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A. Benchmarks Definitions 
A.l List 

object ListBenchmark { 
sealed abstract class List 

case class Cons(head: Int, tail: List) extends List 
case object Nil extends List 

def size(l; List) : Int = (I match { 

case Nil ^ 

case Cons(_, t) => 1 + size(t) 
}) ensuring(res => res > 0) 

def content(l: List): Set[lnt] = I match { 

case Nil => Set.empty[lnt] 

case Cons(i, t) ^ Set(i) ++ content(t) 
} 

def abs(i : Int) : Int = { 

lf(i < 0) -i else i 
} ensuring(_ > 0) 

def insert(inl: List, v: Int) = choose { 
(out : List) ^ 
content(out) == content(inl) ++ Set(v) 

} 

def delete(inl: List, v: Int) = choose { 
(out : List) ^ 
content(out) == content(inl) — Set(v) 

} 

def union(inl: List, in2: List) = choose { 
(out : List) ^ 
content(out) == content(inl) ++ content(in2) 

} 

def diff(inl: List, in2: List) = choose { 
(out : List) ^ 
content(out) == content(inl) — content(in2) 

} 

def split(list : List) : (List, List) = { 
choose { (res : (List, List)) => 
val si — size(res._l) 
val s2 = size(res._2) 

abs(sl - s2) < 1 && si + s2 == size(list) && 
content(res._l) ++ content(res._2) == content(list) 
} 
} 
} 



A.2 SortedList 

object SortedListBenchmark { 
sealed abstract class List 

case class Cons(head: Int, tail: List) extends List 
case object Nil extends List 

def size(l: List) : Int = (I match { 

case Nil ^ 

case Cons(_, t) => 1 + size(t) 
}) ensuring(res => res > 0) 

def content(l: List): Set[lnt] = I match { 
case Nil => Set.empty[lnt] 
case Cons(i, t) => Set(i) ++ content(t) 

} 

def isSorted(list : List) : Boolean = list match { 

case Nil => true 

case Cons(_, Nil) => true 

case Cons(xl, Cons(x2, _)) lf(xl > x2) => false 

case Cons(_, xs) ^ isSorted(xs) 
} 

def insert(inl: List, v: Int) = choose { 
(out : List) ^ 

isSorted(inl) && 

(content(out) == content(inl) ++ Set(v)) && 

isSorted(out) 
} 

def insertAlways(inl: List, v: Int) = choose { 
(out : List) ^ 

isSorted(inl) M 

(content(out) == content(inl) ++ Set(v)) && 

isSorted(out) && 

size(out) == size(inl) + 1 
} 

def delete(inl: List, v: Int) = choose { 
(out : List) ^ 

isSorted(inl) M 

(content(out) == content(inl) — Set(v)) && 

isSorted(out) 
} 

def union(inl: List, in2: List) — choose { 
(out : List) => 

isSorted(inl) && 

isSorted(in2) && 

(content(out) == content(inl) ++ content(in2)) 

isSorted(out) 
} 

def diff(inl: List, in2: List) = choose { 
(out : List) => 
isSorted(inl) && 
isSorted(in2) && 

(content(out) == content(inl) — content(in2)) 
isSorted(out) 
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} 



A.4 UnaryNumerals 



// In order to synthesize insertionSort, we let 
// insert in the scope. Similarly for mergeSort, 
// we keep only split and union in the scope. 
def sort(list: List): List = choose { 
(res: List) => 

isSorted(res) && 

content(res) == content(list) 
} 

} 

A.3 StrictlySortedList 

object Complete { 

sealed abstract class List 

case class Cons(head: Int, tail: List) extends List 

case object Nil extends List 

def size(l: List) : Int = (I match { 

case Nil => 

case Cons(_, t) => 1 + size(t) 
}) ensuring(res ^ res > 0) 

def content(l: List): Set[lnt] = I match { 
case Nil ^ Set.empty[lnt] 
case Cons(i, t) ^ Set(i) ++ content(t) 

} 

def isSorted(list : List) : Boolean = list match { 

case Nil ^ true 

case Cons(_, Nil) => true 

case Cons(xl, Cons(x2, _)) if(xl > x2) ^ false 

case Cons(_, xs) => isSorted(xs) 
} 

def insert(inl: List, v: Int) — choose { 
(out : List) ^ 

isSorted(inl) && 

(content(out) == content(inl) ++ Set(v)) && 

isSorted(out) 
} 

def delete(inl: List, v: Int) = choose { 
(out : List) ^ 

isSorted(inl) && 

(content(out) == content(inl) — Set(v)) && 

isSorted(out) 
} 

def union(inl: List, in2: List) — choose { 
(out : List) ^ 
isSorted(inl) && 
isSorted(in2) && 

(content(out) == content(inl) ++ content(in2)) 
isSorted(out) 
} 
} 



object UnaryNumeralsBenchmark { 
sealed abstract class Num 
case object Z extends Num 
case class S(pred: Num) extends Num 

def value(n:Num) : Int = { 
n match { 
case Z =^ 
case S(p) => 1 + value(p) 

} 
} ensuring (_ > 0) 

def add(x: Num, y: Num): Num = { 

choose { (r : Num) => 
value(r) == value(x) + value(y) 

} 
} 

def distinct(x: Num, y: Num): Num = { 
choose { (r : Num) => 
value(r) != value(x) kk 
value(r) != value(y) 
} 
} 

def mult(x: Num, y: Num): Num = { 
choose { (r : Num) => 
value(r) == value(x) * value(y) 



} 



} 
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